Importing all the libraries¶
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
Loading the dataset¶
In [2]:
dataset = pd.read_csv("C:/Users/Renick Andrews/Downloads/bank-full.csv")
Size of the dataset¶
In [3]:
dataset.shape
Out[3]:
Attribute description¶
In [4]:
dataset.columns
Out[4]:
In [5]:
dataset.head(10)
Out[5]:
Datatypes of the attributes¶
In [6]:
dataset.dtypes
Out[6]:
Finding missing values of the dataset¶
In [7]:
dataset.isnull().sum()
Out[7]:
There are no presence of missing values in the dataset
Five number summary of the dataset¶
In [8]:
dataset.describe().transpose()
Out[8]:
There are negative values present in the pdays column of the dataset .Hence, we replace it to positive values
In [9]:
dataset.pdays=dataset.pdays.replace(-1,1)
In [10]:
dataset.head()
Out[10]:
Visualizing the presence of outliers in the dataset¶
In [11]:
sns.boxplot(dataset['age'])
Out[11]:
There are clients over 70 and there they are outliers.
In [12]:
sns.boxplot(dataset['balance'])
Out[12]:
The balance column is very widely distributed having few clients with a balance of more than 5000 Rupees.
In [13]:
sns.boxplot(dataset['day'])
Out[13]:
There are no outliers the day column.
In [14]:
sns.boxplot(dataset['duration'])
Out[14]:
There are few outliers in this column whose duration goes more than 1000.
In [15]:
sns.boxplot(dataset['pdays'])
Out[15]:
There are outliers in the pdays column having values more than 500
In [16]:
sns.boxplot(dataset['campaign'])
Out[16]:
There are a very few contacts who performed during the campaign although there are contacts who performed.
In [17]:
sns.boxplot(dataset['previous'])
Out[17]:
There is only one contact performed previously and all others fall inside the scale .
Checking the distribution of the dependent column¶
In [18]:
pd.value_counts(dataset["Target"]).plot(kind="bar")
Out[18]:
In [19]:
dataset["Target"].value_counts(normalize=True)
Out[19]:
In [20]:
dataset.groupby(dataset["Target"]).count()
Out[20]:
The customers who fall into the YES category are very less in number than those who told NO. YES clients range upto 5289 and NO clients count upto 39922.
Converting object datatype to categorical for visualization.¶
In [21]:
from sklearn.preprocessing import LabelEncoder
label = LabelEncoder()
dataset['month'] = label.fit_transform(dataset['month'])
dataset['poutcome'] = label.fit_transform(dataset['poutcome'])
dataset['job'] = label.fit_transform(dataset['job'])
dataset['marital'] = label.fit_transform(dataset['marital'])
dataset['education'] = label.fit_transform(dataset['education'])
dataset['default'] = label.fit_transform(dataset['default'])
dataset['contact'] = label.fit_transform(dataset['contact'])
dataset['housing'] = label.fit_transform(dataset['housing'])
dataset['loan'] = label.fit_transform(dataset['loan'])
dataset['Target'] = label.fit_transform(dataset['Target'])
Checking the relationship between data using pairplot.¶
In [22]:
sns.pairplot(dataset, diag_kind="kde")
Out[22]:
There is no strong realtionship between the target and any of the independent variables.Though we can say that the duration column has a slight impact on the target column.
Check the correlation of different datatypes using Heatmap.¶
In [23]:
plt.figure(figsize=(15,15))
sns.heatmap(dataset.corr(),annot=True)
Out[23]:
There is no strong correlation betwwen two variables.
The pdays column and the previous column are midly correlated having the correlation value of 0.45.
The target column and the duration column are midly correlated having the correlation value of 0.39.
The month column and the housingcolumn are slightly correlated having the correlation value of 0.27.
The pdays column and the previous column are midly correlated having the correlation value of 0.45.
The target column and the duration column are midly correlated having the correlation value of 0.39.
The month column and the housingcolumn are slightly correlated having the correlation value of 0.27.
Creation of a dataframe dataframe.¶
In [22]:
dataframe = pd.DataFrame(dataset)
dataframe.head()
Out[22]:
Converting all the categories into numbers which identify a particular using labelencoder.¶
In [23]:
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
dataframe['job'] = le.fit_transform(dataframe['job'])
dataframe['marital'] = le.fit_transform(dataframe['marital'])
dataframe['education'] = le.fit_transform(dataframe['education'])
dataframe['default'] = le.fit_transform(dataframe['default'])
dataframe['contact'] = le.fit_transform(dataframe['contact'])
dataframe['housing'] = le.fit_transform(dataframe['housing'])
dataframe['loan'] = le.fit_transform(dataframe['loan'])
dataframe['month'] = le.fit_transform(dataframe['month'])
dataframe['poutcome'] = le.fit_transform(dataframe['poutcome'])
dataframe['Target'] = le.fit_transform(dataframe['Target'])
In [24]:
dataframe.head()
Out[24]:
Checking the datatypes after conversion.¶
In [25]:
dataframe.dtypes
Out[25]:
All the columns are converted integer datatype but there are few columns which are categorical and each number represents a category in such columns.
Converting the integer datatype to categorical datatype.¶
In [26]:
dataframe.job=dataframe.job.astype('category')
dataframe.education=dataframe.education.astype('category')
dataframe.marital=dataframe.marital.astype('category')
dataframe.default=dataframe.default.astype('category')
dataframe.contact=dataframe.contact.astype('category')
dataframe.housing=dataframe.housing.astype('category')
dataframe.loan=dataframe.loan.astype('category')
dataframe.month=dataframe.month.astype('category')
dataframe.poutcome=dataframe.poutcome.astype('category')
dataframe.Target=dataframe.Target.astype('category')
In [27]:
dataframe.dtypes
Out[27]:
Hence we can see that the datatypes have been converted.
Standardizing the dataset columns using Standardscaler.¶
In [28]:
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler
X_std = pd.DataFrame(StandardScaler().fit_transform(dataframe.values))
X_std.columns = dataframe.columns
Separating the independent variables which is column 0 to column 16 and storing them in X array.
Storing the target variable which is column 17 in Y array
In [29]:
X_dataframe = dataframe.loc[:, dataframe.columns != 'Target']
y_dataframe = dataframe.loc[:, dataframe.columns == 'Target']
Importing few other necessary libraries.¶
In [30]:
from sklearn import model_selection
from sklearn import metrics
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
Splitting the dataset into train and test.¶
In [31]:
features=X_dataframe.iloc[:, 0:16]
features_array = features.values
target_labels = y_dataframe.values
X_train, X_test, y_train, y_test = model_selection.train_test_split(features_array, target_labels, test_size=0.3, random_state=1)
In [32]:
print("{0:0.2f}% data is in training set".format((len(X_train)/len(dataframe.index))*100))
print("{0:0.2f}% data is in test set".format((len(X_test)/len(dataframe.index))*100))
In [33]:
print(X_train.shape)
print(y_train.shape)
The data is split into the ratio of 70:30.
LOGISTIC REGRESSION MODEL¶
In [34]:
model = LogisticRegression()
model.fit(X_train, y_train)
model_score = model.score(X_test, y_test)
y_predict = model.predict(X_test)
print(model_score)
print(metrics.confusion_matrix(y_test, y_predict))
The confusion matrix :
Number of true positives = 11763
Number of false negatives = 1230
Number of true negatives = 321
Number of false positives = 250
The number of correct predictions of identifing potential customers who are relatively more to subscribe term deposit are high i.e., 11763 and,
The customers who would not subscribe but predicted they will are 250.
The customers who would not subscribe but predicted they will not are 321 , which is relatively less than the number of true positives.
The customers who would subscribe but predicted they will not are 1230.
Number of true positives = 11763
Number of false negatives = 1230
Number of true negatives = 321
Number of false positives = 250
The number of correct predictions of identifing potential customers who are relatively more to subscribe term deposit are high i.e., 11763 and,
The customers who would not subscribe but predicted they will are 250.
The customers who would not subscribe but predicted they will not are 321 , which is relatively less than the number of true positives.
The customers who would subscribe but predicted they will not are 1230.
In [35]:
print(metrics.classification_report(y_test, y_predict))
The accuracy of this model is 89%.
The precision of this model is 73%.
The recall of this model is 59%.
The precision of this model is 73%.
The recall of this model is 59%.
NAIVE BAIYES MODEL¶
In [36]:
model = GaussianNB()
model.fit(X_train, y_train)
predictions=model.predict(X_test)
print(model_score)
print(metrics.confusion_matrix(y_test,predictions))
The confusion matrix :
Number of true positives = 10705
Number of true negatives = 701
Number of false positives = 1308
Number of false negatives = 850
The number of correct predictions of identifing potential customers who are relatively more to subscribe term deposit are high i.e., 11757 and,
The customers who would not subscribe and predicted they will not are 701, which is relatively less than the number of true positives
The customers who would not subscribe but predicted they will are 1308.
The customers who would subscribe but predicted they will not are 850.
Number of true positives = 10705
Number of true negatives = 701
Number of false positives = 1308
Number of false negatives = 850
The number of correct predictions of identifing potential customers who are relatively more to subscribe term deposit are high i.e., 11757 and,
The customers who would not subscribe and predicted they will not are 701, which is relatively less than the number of true positives
The customers who would not subscribe but predicted they will are 1308.
The customers who would subscribe but predicted they will not are 850.
In [37]:
expected = y_test
predicted = model.predict(X_test)
print(metrics.classification_report(expected, predicted))
The accuracy of this model is 84%.
The precision of this model is 64%.
The recall of this model is 67%.
The precision of this model is 64%.
The recall of this model is 67%.
KNN MODEL¶
In [38]:
from sklearn.neighbors import KNeighborsClassifier
NNH = KNeighborsClassifier(n_neighbors= 3 , weights = 'distance')
NNH.fit(X_train, y_train)
Out[38]:
In [39]:
predicted_labels = NNH.predict(X_test)
In [40]:
print(model_score)
print(metrics.confusion_matrix(y_test, predicted_labels))
Number of true positives = 11430
Number of true negatives = 465
Number of false positives = 583
Number of false negatives = 1086
Number of true negatives = 465
Number of false positives = 583
Number of false negatives = 1086
The number of correct predictions of identifing potential customers who are relatively more to subscribe term deposit are high i.e., 11430 and,
The customers who would not subscribe but predicted they will not are 465 , which is relatively less than the number of true positives.
The customers who would not subscribe but predicted they will are 583.
The customers who would subscribe but predicted they will not are 1086.
The customers who would not subscribe but predicted they will not are 465 , which is relatively less than the number of true positives.
The customers who would not subscribe but predicted they will are 583.
The customers who would subscribe but predicted they will not are 1086.
In [41]:
print(metrics.classification_report(y_test, predicted_labels))
The accuracy of this model is 88%.
The precision of this model is 68%.
The recall of this model is 63%.
The precision of this model is 68%.
The recall of this model is 63%.
After observing the performance of all the models, the logistic regression model is the best model because the accuracy,precision,recall,f1-score and the confusion matrix give out better values than the naive bayes and the knn models.
Building the Ensemble models.¶
Building a decision tree usingthe GINI criterion.¶
In [42]:
dTree = DecisionTreeClassifier(criterion = 'gini', random_state=1)
dTree.fit(X_train, y_train)
Out[42]:
In [43]:
print(dTree.score(X_train, y_train))
print(dTree.score(X_test, y_test))
Perfomance on training data using gini criterion = 100%.
Perfomance on test data using gini criterion = 87.00%.
Perfomance on test data using gini criterion = 87.00%.
Building a decision tree using the ENTROPY criterion.¶
In [44]:
model_entropy=DecisionTreeClassifier(criterion='entropy')
In [45]:
model_entropy.fit(X_train, y_train)
Out[45]:
In [46]:
print(model_entropy.score(X_train, y_train))
print(model_entropy.score(X_test, y_test))
Perfomance on training data using gini criterion = 100%.
Perfomance on test data using gini criterion = 87.76%.
Perfomance on test data using gini criterion = 87.76%.
The Entropy Criteria is better to evaluate the Decision tree because its perfomance on the test data is little better than that of the Gini Criteria.
In [47]:
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from IPython.display import Image
from sklearn.tree import export_graphviz
from sklearn.externals.six import StringIO
from sklearn import tree
from os import system
import pydotplus
import graphviz
Visualizing the decision tree.¶
In [48]:
X_train, X_test, y_train, y_test = train_test_split(X_dataframe, y_dataframe, test_size=.30, random_state=1)
train_char_label = ['No', 'Yes']
dataframe_Tree_File = open('df_tree.dot','w')
dot_data = tree.export_graphviz(model_entropy, out_file=dataframe_Tree_File, feature_names = list(X_train), class_names = list(train_char_label))
dataframe_Tree_File.close()
Hyper Key Parameters:
model_entropy- The best criteria out of gini and entropy
out_file- The output file, which we initialized before
(Optional)
feature_names- list of strings
class_names- labels- we made it Yes and No
model_entropy- The best criteria out of gini and entropy
out_file- The output file, which we initialized before
(Optional)
feature_names- list of strings
class_names- labels- we made it Yes and No
Checking if there is a error while we output the tree¶
In [50]:
retCode = system("dot -Tpng df_tree.dot -o df_tree.png")
if(retCode>0):
print("system command returning error: "+str(retCode))
else:
display(Image("df_tree.png"))
Regularization (REDUCING OVER FITTING)¶
In [51]:
dTree = DecisionTreeClassifier(criterion = 'gini', max_depth = 3, random_state=1)
dTree.fit(X_train, y_train)
print(dTree.score(X_train, y_train))
print(dTree.score(X_test, y_test))
In [52]:
train_char_label = ['No', 'Yes']
df_Tree_File = open('df_treeR.dot','w')
dot_data = tree.export_graphviz(dTree, out_file=df_Tree_File, feature_names = list(X_dataframe), class_names = list(train_char_label))
df_Tree_File.close()
retCode = system("dot -Tpng df_treeR.dot -o df_treeR.png")
if(retCode>0):
print("system command returning error: "+str(retCode))
else:
display(Image("df_treeR.png"))
In [53]:
print (pd.DataFrame(dTree.feature_importances_, columns = ["Imp"], index = X_dataframe.columns))
To check the score and print the heatmap¶
In [55]:
print(dTree.score(X_test , y_test))
y_predict = dTree.predict(X_test)
cm=metrics.confusion_matrix(y_test, y_predict, labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g')
Out[55]:
Ensemble Learning - Bagging¶
In [56]:
from sklearn.ensemble import BaggingClassifier
bgcl = BaggingClassifier(base_estimator=dTree, n_estimators=50,random_state=1)
bgcl = bgcl.fit(X_train, y_train)
In [57]:
y_predict = bgcl.predict(X_test)
print(bgcl.score(X_test , y_test))
cm=metrics.confusion_matrix(y_test, y_predict,labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g')
Out[57]:
Note: n_estimators= 50 gives the best result.
Ensemble Learning - AdaBoosting¶
In [58]:
from sklearn.ensemble import AdaBoostClassifier
abcl = AdaBoostClassifier(n_estimators=50, random_state=1)
abcl = abcl.fit(X_train, y_train)
In [59]:
y_predict = abcl.predict(X_test)
print(abcl.score(X_test , y_test))
cm=metrics.confusion_matrix(y_test, y_predict,labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g')
Out[59]:
Ensemble Learning - GradientBoost¶
In [60]:
from sklearn.ensemble import GradientBoostingClassifier
gbcl = GradientBoostingClassifier(n_estimators = 50,random_state=1)
gbcl = gbcl.fit(X_train, y_train)
In [61]:
y_predict = gbcl.predict(X_test)
print(gbcl.score(X_test, y_test))
cm=metrics.confusion_matrix(y_test, y_predict,labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g')
Out[61]:
Ensemble RandomForest Classifier¶
In [62]:
from sklearn.ensemble import RandomForestClassifier
rfcl = RandomForestClassifier(n_estimators = 50, random_state=1,max_features=12)
rfcl = rfcl.fit(X_train, y_train)
In [63]:
y_predict = rfcl.predict(X_test)
print(rfcl.score(X_test, y_test))
cm=metrics.confusion_matrix(y_test, y_predict,labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g')
Out[63]:
